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A key benefit of the regression-discontinuity design is that it effectively assesses 
the extent that developmental programs result in improving student retention 
and academic success. 



Should the nation’s colleges and uni- 
versities continue to fund and support 
developmental programs? This is a key 
policy question given the grim financial 
scenario at many institutions and the 
fact that developmental programs are 
often perceived as putting a signifi- 
cant drain on an often limited amount 
of resources (Cloud, 2002; Institute 
for Higher Education, 1998; Lazarick, 
1997). Nonetheless, many institutions 
lack a true understanding as to whether 
developmental programs are effective 
in helping students persist (O’Hear & 
MacDonald, 1995a, 1995b). This brief 
provides institutional and state policy 
makers with a methodological tool, 
the regression-discontinuity design, 
to effectively assess the extent that 
developmental programs result in 
improving student retention and aca- 
demic success. 

This brief presents the results of a 
study which found that a developmental 



mathematics program at a four-year 
institution is an effective mechanism 
for retaining students (Lesik, 2007). 
This study employed a regression- 
discontinuity model that researchers 
can use to establish a causal link 
between participation in a develop- 
mental program and student retention. 
Because a causal inference can be 
made, policy makers have an opportu- 
nity to base their decisions regarding 
the effectiveness of developmental 
programs strictly on the merits of the 
program, without bias. The basic theory 
behind the regression-discontinuity 
design will be described along with the 
findings. Finally, this brief closes with 
some implications for how the regres- 
sion-discontinuity design can be a 
crucial tool for policy makers who are 
interested in making data-informed 
decisions about the effectiveness of 
their developmental programs, and how 
using such a research design can shape 



educational policy and assess measur- 
able outcomes 

The National Center for Educational 
Statistics (2003) found that almost 
one-third of all entering college stu- 
dents are not academically prepared 
for college-level work, and therefore 
need some form of support to help 
them gain the skill-base necessary to 
succeed in college. Because so many 
students are entering college without 
the needed skill-base, developmental 
programs have become an integral part 
of the undergraduate curriculum at 
many institutions (Boylan, Bonham, & 
White, 1999; Jenkins & Boswell, 2002). 
The main purpose of developmental 
programs is to help underprepared stu- 
dents achieve their maximum potential 
by providing the necessary academic 
support services that are designed to 
improve basic skill competencies in 
subjects such as reading, writing, and 
mathematics (Lazarick, 1997). Devel- 



opmental programs can help students 
who might not have the opportunity 
to earn a college degree by providing 
another chance to gain the skill-base 
needed to persist and succeed in col- 
lege (Boylan, Bonham, & Bliss, 1994; 
Kulik, Kulik, & Shwalb, 1983). 

Controversies surrounding 
developmental education 
Using data to inform policy decisions 
regarding developmental programs in 
higher education is one of the top pri- 
orities on the agendas of many college 
and university administrations. On 
the one hand, many colleges and uni- 
versities are facing budget cuts and 
demanding higher standards. On the 
other hand, developmental programs 
can be a necessity because there are 
so many academically underprepared 
students entering college. Since 
developmental programs can serve 
as a bridge that aligns high school 
preparation and college expectations, 
underprepared students can be given 
another opportunity to become ready 
to pursue college-level work (Lazarick, 



1997). Furthermore, community colleges 
that have "open-door" admissions poli- 
cies rely on developmental programs 
to fulfill their mission, and such pro- 
grams can create a path of access to 
college for students who may not have 
the initial skill-base needed to per- 
sist through college (Testone, 2001). 
Effectively assessing developmental 
programs is necessary in order to bal- 
ance the commitment to funding such 
programs, while also giving under- 
prepared students the opportunity to 
build their skills in order to succeed 
in college and eventually earn a col- 
lege degree. 

With such a strong presence of 
developmental programs at many insti- 
tutions, little is known about whether 
these programs can be an effective 
tool that helps underprepared students 
stay enrolled in college and ultimately 
earn a college degree (Moore, Jensen, 

& Hatch, 2002). Balancing the need to 
commit a substantial amount of human 
and financial resources, while also pro- 
viding underprepared students with the 
necessary support to build their skill- 



base, has been an insurmountable chal- 
lenge to many colleges and universities 
(0’Hear& MacDonald, 1995a, 1995b). 

In fact, many four-year institutions do 
not even offer developmental programs 
because of their high cost and because 
some administrators do not believe that 
they are beneficial to their students 
(Jenkins & Boswell, 2002). 

Given that developmental pro- 
grams are such a controversial topic in 
higher education, numerous research 
studies have been conducted to deter- 
mine whether these programs can be 
an effective mechanism for retaining 
underprepared students (e.g. Feldman, 
1993; Gates & Creamer, 1984; Hardin, 
1998; Institute for Higher Education, 
1998; Jacob & Lefgren, 2002; Snyder, 
Hackett, Stewart, & Smith, 2003). 
Despite political controversies and 
countless numbers of studies looking 
for ways to assess developmental 
programs, policy makers must filter 
through mounds of research hoping 
to gain some insight as to whether such 
programs are truly beneficial in keeping 
students in college. One of the biggest 
questions that policy makers face is 
whether their institution should even 
offer developmental programs, espe- 
cially if they are uncertain about their 
effectiveness (Cloud, 2002; Jenkins & 
Boswell, 2002). 

Limitations of traditional 
regression analysis 
As policy makers look to make data- 
informed decisions, they rely on 
researchers to accurately quantify a 
phenomenon of interest. Researchers 
use statistical methods and techniques 
to aid them in making such quanti- 
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By relying on more tra- 
ditional methods, policy 
makers must make deci- 
sions based on analytical 
techniques that can only 
include observed variables, 
and thus can not assess 
the true impact that a 
developmental program 
has on student retention. 

fications. Researchers interested in 
assessing the effect that developmental 
programs have on student retention 
typically develop some form of a mul- 
tiple regression model by using either 
ordinary least squares or logistic regres- 
sion analysis (e.g. Hoyt, 1999; Snyder et 
al., 2003). To model student retention, 
researchers can use regression analysis 
to develop a model that describes all 
the different factors that could impact 
whether a student remains enrolled in 
college over a fixed period of time. In 
order to determine the effect of partici- 
pating in a developmental program, a 
dichotomous variable is usually included 
in the regression model as a way to rep- 
resent whether a student participated 
in a developmental program along with 
a vector of observable control variables. 
Then, based on the results of the regres- 
sion analysis, researchers will make 
conclusions about whether they believe 
the developmental program is effective 
in keeping students in college by inter- 
preting the estimate of the coefficient of 
the dichotomous treatment variable. 

Although typical regression mod- 
eling strategies such as logistic regres- 
sion and ordinary least squares can be 



useful in gaining a basic understanding 
of the relationship between partici- 
pating in a developmental program and 
student retention, a cause and effect 
relationship can not be inferred. The 
reason a cause and effect relationship 
can not be inferred is because there 
is no way to control for every possible 
factor that could impact student reten- 
tion. Furthermore, it can also be diffi- 
cult to establish whether the measures 
used for the control variables are even 
reliable. Thus, it is virtually impossible 
to observe and accurately measure 
every single factor that could impact 
student retention and include such 
factors in a regression model. Short of 
performing a true random experiment, 
it is difficult to establish a cause and 
effect relationship between partici- 
pating in a developmental program and 
student retention. It is no surprise that 
the findings of many research studies 
on the effectiveness of developmental 
programs at comparable institutions 
tend to be mixed and inconclusive. 

Another limitation of many studies 
investigating the impact that devel- 
opmental programs have on student 
retention is that many of these studies 
rely on statistical methods for cross- 
sectional data and do not model 
retention patterns over time. Whether 
students drop out of college at some 
fixed point in their college career is 
clearly of interest to policy makers. 

Yet, the timing of when a student is 
most at risk for dropping out of college 
for the first time can also be revealing 
(Singer & Willett, 2003). Knowing when 
a student is most at risk for drop- 
ping out of college for the first time 
can provide the faculty and staff at 



the institution with an opportunity to 
develop interventions based on those 
crucial points in time. 

Short of performing a true random 
experiment, it can be difficult to isolate 
the causal effect of participating in a 
developmental program, and most 
research designs do not account for the 
longitudinal nature of student dropout 
data. Because of these two limitations, 
any policy decisions based on the 
results of research studies that use 
traditional regression modeling strate- 
gies that do not take into account the 
longitudinal nature of student retention 
could be misguided. By relying on more 
traditional methods, policy makers must 
make decisions based on analytical tech- 
niques that can only include observed 
variables, and thus can not assess the 
true impact that a developmental pro- 
gram has on student retention. 

The regression-discontinuity design 
The regression-discontinuity design 
is a simple extension of regression 
analysis (Thistlethwaite & Campbell, 
1960). The difference between using 
the regression-discontinuity design 
and traditional regression analysis 
is that the regression-discontinuity 
design relies on an exogenous, pre- 
determined assignment variable 
with a fixed cutoff score (Reichardt, 
Trochim, & Cappelleri, 1995; Shadish, 
Cook, & Campbell, 2002; Trochim, 
1984). Given an exogenous assign- 
ment variable, treatment and control 
groups can be established that are 
equivalent to each other and similar 
to what a true random experiment 
would generate (Shadish et al., 2002; 
vanDerKlaauw, 2002). 
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An exogenous assignment vari- 
able is readily available, since most 
institutions regularly use some form 
of a diagnostic test to gauge students’ 
entering skill levels (e.g. Leake & Lesik, 
2007; Lesik, 2006, 2007). Such a test is 
usually administered prior to when the 
students enroll at the university (Boylan 
et al., 1999). A diagnostic test can serve 
as an exogenous assignment variable 
provided that a uniform cutoff score 
is established and adhered to for all 
students and that the test is given prior 
to receiving treatment (Shadish et al., 
2002; Trochim, 1984). The cutoff score 
of the assignment variable can be used 
to partition the sample into those stu- 
dents who may be able to benefit from 
participating in the developmental 
program and those students whose 
skill-base is appropriate for them to 
begin college-level work (Leake & 

Lesik, 2007; Lesik, 2006, 2007; Shadish 
et al., 2002; vanDerKlaauw, 2002). 

The institution in this study relied 
on faculty members in the mathematics 
department to align the material on 
the diagnostic test with the content of 
the mathematics courses offered at 
the university. This allowed for a spe- 
cific cutoff score on the diagnostic test 
to be established. Students who scored 
below the cutoff score were assigned 
to the developmental mathematics 
program because they did not show 
that they had the skill-base neces- 
sary to succeed in college. Students 
who scored above the cutoff were not 
so assigned because their test scores 
indicated that they had the necessary 
skill-base. 

The region on either side of the 
cutoff score of the assignment variable 

4 



The most significant policy 
impact from using the 
regression-discontinuity 
design is that the causal 
effect of participating in 
a developmental program 
on student retention can 
be established. 

is of great interest because it is within 
this region where the regression- 
discontinuity design can emulate a 
random experiment (Pettersson- 
Lidbom, 2003; Shadish et al., 2002; 
Trochim, 1984; vanDerKlaauw, 2002). 
If a predetermined cutoff score on 
an exogenous assignment variable is 
established and individuals who fall 
just above the cutoff are compared to 
those individuals who fall just below 
the cutoff, then these individuals 
should be similar to each other in 
every way except for the assignment 
to either the treatment or control 
group (Leake & Lesik, 2007; Lesik, 
2006, 2007; vanDerKlaauw, 2002). 

Given such an assignment variable, 
it is simple to run a regression- 
discontinuity analysis with virtually 
any type of regression model, and 
the interpretations are similar (i.e. 
Berk & DeLeeuw, 1999; Berk & 
Rauma, 1983; Leake & Lesik, 2007; 
Lesik, 2006, 2007; Moss & Yeaton, 
2006). The only difference in using 
the regression-discontinuity design 
as compared to using traditional 
regression models is that the 
assignment variable is included as a 
covariate in a regression model along 
with the binary treatment indicator 
(Shadish et al., 2002; Trochim, 1984). 



The regression-discontinuity 
design can be incorporated within 
virtually any type of regression mod- 
eling strategy such as ordinary least 
squares (e.g. Leake & Lesik, 2007; 
Moss & Yeaton, 2006), logistic regres- 
sion (Berk & DeLeeuw, 1999; Berk & 
Rauma, 1983; Lesik, 2006), or as in the 
case for the study described herein, 
discrete-time survival analysis (Lesik, 
2007). However, it can be a difficult chal- 
lenge to identify and address any threats 
that may introduce bias in the estimate 
of the program effect when using the 
regression-discontinuity design. While it 
is true that the regression-discontinuity 
design is easy to incorporate within vir- 
tually any regression model, obtaining 
a valid estimate when using the design 
lies at the heart of the work and remains 
the biggest challenge.' 

Background of the study 
The institution considered in this study 
is a large state university located in the 
northeast United States. The develop- 
mental mathematics program at this 
institution provides support to those 
students who enter the university who 
are identified as not having the skill- 
base needed to succeed in college 
level mathematics. The objective of 
the Lesik (2007) study was to deter- 
mine if students who participated in 
a developmental mathematics pro- 
gram were more likely to persist over 
the course of their first three years in 
college as compared to nondevelop- 
mental students. 

’ For a detailed description of how to address common 
threats to validity when using a regression-discontinuity 
analysis in practice, see Lesik (20081. 
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Figure 1 : Fitted hazard function at the cutoff score of the assignment 
variable using a regression-discontinuity hazard model. 
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Nondevelopmental - - - - Developmental 



Source: This figure is reproduced with the kind permission of Springer Science and Business Media from Research 
In Higher Education, 4S(5), 2007, Figure 3, page 602, Do Developmental Mathematics Programs have a Causal Impact 
on Student Retention? An Application of Discrete-Time Survival and Regression-Discontinuity Analysis by Sally A. Lesik. 



The developmental mathematics 
program at this institution consisted of 
a single stand-alone course in interme- 
diate algebra. Entering students were 
assigned to the course based on the 
score that they received on a diagnostic 
placement examination in mathematics 
taken prior to entering the university 
for the first time. The cutoff score for 
the examination was determined by 
the faculty of the mathematics depart- 
ment, and it was based on the level of 
difficulty of the questions given in the 
diagnostic test. Students who scored 
above the cutoff score were identified 
as having the necessary foundation in 
mathematics needed in college and 
were not assigned to the develop- 
mental mathematics course. 



Findings 

Figure 1 presents a close-up of the 
dropout profile at the centered cutoff 
score of 0. This graph shows the risk 
of dropout (y-axis) over the course of 
six semesters (x-axis) at the cutoff 
score of 0. The dashed line repre- 
sents the dropout profile for those 
students who participated in the 
developmental course, and the solid 
line represents the dropout profile for 
those students who did not partici- 
pate in the developmental course. It 
is at the centered cutoff score where 
the students who participated in the 
developmental course and the stu- 
dents who did not participate in the 
developmental course were equiva- 
lent in every respect. The difference 
in the risk of dropout is significant 



at the 0.05 level (Lesik, 2007). These 
findings suggested that students who 
did not participate in the develop- 
mental program were approximately 
four times more likely to drop out of the 
university over the course of their first 
three years as compared to equivalent 
students who did participate in the 
developmental program. 

Policy impact 

If the true effectiveness of a develop- 
mental program cannot be established, 
then policy makers may decide that 
the human and financial resources 
that these programs demand are not 
being used wisely. However, the dis- 
continuation of such programs based 
on perceptions of them being ineffec- 
tive would cause a large number of 
students to miss out on an opportunity 
to earn a college degree if they are not 
given the chance to gain the skill-base 
necessary to succeed. Unless policy 
makers can obtain a true and unbiased 
estimate of whether their develop- 
mental programs are effective in 
helping underprepared students per- 
sist in college, then valuable resources 
may be diverted to other programs that 
may not help this given population. 

The most significant policy impact 
from using the regression-discontinuity 
design is that the causal effect of par- 
ticipating in a developmental program 
on student retention can be estab- 
lished. Provided that the assignment 
to the treatment program is based on 
an exogenous assignment variable and 
that any threats to validity are identi- 
fied and addressed appropriately, it is 
unlikely that any other factors could 
impact the estimate of the treatment 
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effect (Leake & Lesik, 2007; Lesik, 
2006,2007). 

Using the regression-discontinuity 
design for evaluating developmental 
programs clearly has advantages over 
using traditional regression techniques. 
This is because the regression- 
discontinuity design emulates a random 
experiment at the cutoff score, and 
thus all possible factors that could 
impact the outcome measure will be 
insignificant. Policy makers can make 
decisions based on the merits of the 
developmental program itself, and not 
be concerned that any other mitigating 
factors may be having an impact on 
the estimate of the treatment effect. 
Thus, the regression-discontinuity 
design can provide researchers and 
policy makers with a valid assessment 
measure of the effectiveness of their 
developmental programs. 

The findings from the regression- 
discontinuity analysis can be used to 
establish new policies and direct new 
and different evaluations. For instance, 
because participating in the program 
increases student retention, policy 
makers at the institution could con- 
sider raising the cutoff score so that 
more students will have the opportu- 
nity to participate in the course, and 
perhaps stay in college longer. By 
raising the cutoff score, policy makers 
may be able to increase student reten- 
tion at their university. However, the 
drawback to implementing such a 
policy is that if the cutoff score was 
raised too high, there might be little 
effect on retention due to the content 
of the current developmental course 
not being aligned with the material 
on the placement examination. 

6 



Regression-discontinuity 
design challenges 
One challenge when evaluating the 
findings from a regression-discontinuity 
design is to determine what specific 
aspects of the developmental math- 
ematics program may cause students 
to persist. Are students persisting 
because they are developing a stronger 
foundation in mathematics? Or did the 
students become more connected with 
the university through their participa- 
tion in the developmental course and 
decide to stay enrolled? One way to 
determine whether it is the actual 
content of the developmental math- 
ematics course that contributes to 
increased retention is to run new 
and different evaluations to see if 
students who participated in the 
developmental program did better 
in their first college-level mathe- 
matics course. For instance, the 
Lesik (2006) study found that students 
who participated in the developmental 
mathematics program also did better 
in their first college-level mathematics 
course as compared to nondevelop- 
mental students. 

In addition to evaluating content 
factors of the program, faculty and 
administrators at the site met to share 
their opinions on why they believed the 
program was effective. Many attributed 
the success of the program to students 
gaining proficiency in mathematics. 

It was also believed that the program 
encouraged students to make use of 
the many different support services 
available at the university. Informally, 
the evaluation was conducted by giving 
a survey to developmental and non- 
developmental students asking them 



to respond to questions regarding their 
knowledge and use of the different sup- 
port services available at the university. 
These findings suggested that develop- 
mental students were better informed 
about the support services offered 
at the university, and more develop- 
mental students regularly used 
such services as compared to non- 
developmental students. 

Summary 

The regression-discontinuity design 
can be an invaluable tool that policy 
makers can use to evaluate their 
developmental programs based solely 
on the merits of the program without 
bias. Because a regression-discontinuity 
analysis emulates a random experiment, 
policy makers can determine whether 
the investment in their developmental 
program is worth the return. Further- 
more, by establishing whether the 
developmental program has a causal 
impact on student retention, then 
policies can be developed and further 
evaluations can be conducted to try to 
isolate the specific aspects of the pro- 
gram that may be causing the effect. 
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